TITLE OF THE INVENTION 
IMAGING APPARATUS AND METHOD 

BACKGROUND OF THE INVENTION 
FIELD OF THE INVENTION 

The present invention relates to an imaging 
apparatus and method, and its control method and, more 
particularly, to an imaging apparatus such as a digital 
camera which saves a photographed image as an electrical 
signal in the apparatus, and an imaging method. 

DESCRIPTION OF THE RELATED ART 

Along with developments of imaging processing 
techniques in recent years, imaging apparatuses such as 
digital still cameras that use these techniques have 
been developed remarkably. Fig. 18 shows the arrangement 
of a digital still camera, and its operation will be 
briefly described below. 

Referring to Fig. 18, reference numeral 91 denotes 
a camera operation unit, which includes a main switch, 
release switch, and the like of a camera. The camera 
operation unit 91 is operated by the user, and an 
overall controller 80 detects a change in state of the 
main switch, and starts power supply to other building 
blocks . 

An object image within the photographing range is 



formed on an imaging unit 82 via a main photographing 
optical system 81, and an electrical signal output from 
the imaging unit 82 is converted into a predetermined 
digital signal via an A/D (Analog to Digital) converter 
83 in turn in units of pixels. The converted digital 
signal is input to a process controller 84. The process 
controller 84 generates R, G, and B color signals on the 
basis of the pixel data sent from the imaging unit 82 . 
Before normal photographing, these color signals are 
transferred to a video memory 89 via a memory controller 
85 periodically, e.g., in units of frames, thus making a 
finder display or the like using a monitor display 90. 

When the user photographs by operating the camera 
operation unit 91, image data for one frame sent from 
the process controller 84 is stored in a frame memory 86 
in response to a control signal from the overall 
controller 80, and the data in the frame memory 86 is 
compressed by the memory controller 85 and a work memory 
87 on the basis of a predetermined compression format. 
The compressed data is stored in an external memory 88 
comprising a nonvolatile memory such as a flash memory 
or the like. According to the advance of the imaging 
apparatus development technique in recent years, a 
camera capable of selectively obtaining 

higher-resolution image data in addition to the normal 
photographing is available. Such high-resolution image 



data is compressed in the predetermined format in the 
same manner as above, and is stored in the external 
memory 88. 

When the user observes already photographed image 
data, the compressed image data stored in the external 
memory 88 is expanded to normal photographed image data 
by the memory controller 85, and the expanded data is 
transferred to the video memory 89. In this way, the 
user can observe the photographed image via the monitor 
display 90. 

As described above, in the digital still camera, 
when photographed image data is stored in a memory or 
the like, the image data is stored after it is 
compressed in a specific compression format or at a 
specific compression rate irrespective of its 
photographing conditions or photographing scene, or is 
compressed by a compression method or at a compression 
rate set by the user himself or herself. 

However, in the digital still camera, since the 
photographed image data is uniformly saved by the camera 
irrespective of its contents, image data photographed in 
an optimal state (compression format) is not always 
saved. The same applies to a case wherein photographing 
for obtaining high-resolution image data is made, i.e., 
when the size of image data becomes large. 

Even when the user himself or herself can set the 



saving method of the photographed image in advance, it 
is very cumbersome for the user to appropriately set the 
image compression method in correspondence with the 
intended photographed image. Hence, if the user has to 
set the saving method every time he or she takes 
photographs, he or she cannot immediately take 
photographs when the need for it arises, and misses a 
so-called shutter chance. Also, when an image 
photographing method must be set in addition to the 
saving method of the photographed image, such operations 
are harder except for expert users. 

Along with advance of the digital technique in 
recent years, many imaging apparatuses capable of 
digital recording, i.e., so-called digital cameras have 
been proposed. Some of these digital cameras comprise a 
compact display device such as a finder or a TFT (Thin 
Film Transistor) liquid crystal display device for 
displaying an image to be photographed or a photographed 
image, a camera shake correction device for correcting a 
camera shake, and a large-capacity storage device such 
as a flash memory or the like for saving JPEG (Joint 
Photographic Experts Group) -compressed images. 

The resolution of an image photographed by the 
above-mentioned digital camera is determined by the type 
of CCD used in the digital camera. Hence, in order to 
photograph a high-resolution image, a CCD having many 



light -receiving elements, e.g., million pixels or more, 
is required, resulting in an expensive digital camera. 

On the other hand, a digital camera having a CCD 
with a small number of pixels improves the resolution by 
interpolation. Since the pixel value obtained by 
interpolation is the average value of surrounding pixel 
values, no problem is posed in a region with relatively 
flat gradation characteristics, but an image edge blurs 
in an image edge region where the gradation 
characteristics change largely. 

SUMMARY OF THE INVENTION 
The present invention has been made to solve the 
aforementioned problems, i.e., to relieve the user of 
cumbersome setups of the image photographing method and 
the saving method of photographed image data, and has as 
its object to provide an imaging apparatus and method, 
which can set the photographing method and image saving 
method in correspondence with the currently selected 
photographing mode. 

In order to achieve the above object, a preferred 
embodiment of the present invention discloses an imaging 
apparatus which comprises an imaging unit for forming an 
object image, and generating an image signal by 
photoelectric conversion, an optical shift unit for 
optically shifting an imaging position of the object 



image in the imaging unit, generation means for 
generating a single image from a plurality of images 
obtained by the imaging unit by a plurality of shifts 
using the optical shift unit, and storage means for 
converting an image obtained by the imaging unit or the 
image generated by the generation means into a 
predetermined data format, and storing the converted 
image in a storage medium, comprising: an operation unit 
for manually setting a photographing mode of the imaging 
apparatus; and control means for setting the data format 
and controlling supply of an image to the storage means 
in correspondence with the set photographing mode. 

It is another object of the present invention to 
provide an imaging apparatus and method, which can set 
the photographing method and image saving method in 
correspondence with the state of the apparatus upon 
photographing . 

In order to achieve the above object, a preferred 
embodiment of the present invention discloses an imaging 
apparatus which comprises an imaging unit for forming an 
object image, and generating an image signal by 
photoelectric conversion, an optical shift unit for 
optically shifting an imaging position of the object 
image in the imaging unit, photographing means for 
storing, in a memory, a plurality of images obtained by 
the imaging unit by a plurality of shifts using the 



optical shift unit, generation means for generating a 
single image from the plurality of images stored in the 
membry, and storage means for converting the image 
stored in the memoiry or the image generated by the 
generation means into a predetermined data format, and 
storing the converted image in a storage medium, 
comprising: a sensor for detecting a state of the 
apparatus upon photographing; and control means for 
setting the data format and controlling supply of an 
image to the storage means in correspondence with the 
detected state of the apparatus. 

It is still another object of the present invention 
to provide an imaging apparatus and method, which can 
set the photographing method and image saving method in 
correspondence with the free capacity of a storage 
medium that saves photographed images . 

In order to achieve the above object, a preferred 
embodiment of the present invention discloses An imaging 
apparatus which comprises an imaging unit for forming an 
object image, and generating an image signal by 
photoelectric conversion, an optical shift unit for 
optically shifting an imaging position of the object 
image in the imaging unit, generation means for 
generating a single image from a plurality of images 
obtained by the imaging unit by a plurality of shifts 



using the optical shift unit, and storage means for 
converting an image obtained by the imaging unit or the 
image generated by the generation means into a 
predetermined data format, and storing the converted 
image in a storage medium, comprising: an operation unit 
for manually setting a photographing mode of the imaging 
apparatus; and control means for setting the data format 
and controlling supply of an image to the storage means 
in correspondence with the set photographing mode, 
wherein the control means sets the data format and 
controls supply of an image to the storage means in 
correspondence with a free storage capacity of the 
storage medium. 

It is still another object of the present invention 
to provide an imaging apparatus and method which can 
capture a high-resolution image with low cost by setting 
a photographing method that corresponds to the 
characteristics of an object. 

In order to achieve the above object, a preferred 
embodiment of the present invention discloses an imaging 
apparatus which comprises an imaging unit for forming an 
object image, and generating an image signal by 
photoelectric conversion, an optical shift unit for 
optically shifting an imaging position of the object 
image in the imaging unit, generation means for 
generating a single image from a plurality of images 



obtained by the imaging unit by a plurality of shifts 
using the optical shift unit, and storage means for 
converting an image obtained by the imaging unit or the 
image generated by the generation means into a 
predetermined data format, and storing the converted 
image in a storage medium, comprising: detection means 
for detecting characteristics of the object; and control 
means for setting a shift method of photographing using 
the optical shift unit in accordance with the detected 
characteristics of the object. 

Other features and advantages of the present 
invention will be apparent from the following 
description taken in conjunction with the accompanying 
drawings, in which like reference characters designate 
the same or similar parts throughout the figures 
thereof . 

BRIEF DESCRIPTION OF THE DRAWINGS 
Fig. 1 is a block diagram showing the arrangement 

of a digital still camera according to the first 

embodiment of the present invention; 

Fig. 2 is a view for explaining the interpolation 

process in the first embodiment; 

Fig . 3 is a flow chart showing the camera sequence 

of the first embodiment; 



Fig, 4 is a flow chart showing the operation in 
photographing- storage mode 1 of the first embodiment; 

Figs . 5A and 5B are flow charts showing the 
operation in photographing- storage mode 2 in the first 
5 embodiment ; 

Fig. 6 is a view for explaining lossless 
compression of a photographed image in the first 
embodiment ; 

Fig. 7 is a view for explaining the principle of 
10 pixel shift photographing in the first embodiment; 

Fig. 8 is a chart for explaining the movement of a 
correction optical system upon pixel shift photographing 
in the first embodiment; 

Fig. 9 is a view for explaining the interpolation 
15 process upon pixel shift photographing in the first 
embodiment ; 

Fig. 10 is a view for explaining lossy compression 
of a photographed image in the first embodiment; 

Fig. 11 is a flow chart showing the camera sequence 
20 according to the second embodiment of the present 
invention; 

Figs. 12 and 13 are flow charts showing the 
operation in photographing- storage mode 3 in the second 
embodiment ; 

25 Figs. 14A and 14B are flow charts showing the 

operation in photographing- storage mode 4 in the second 
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embodiment ; 

Figs. 15A and 15B are flow charts showing the 
camera sequence according to the third embodiment of the 
present invention; 
5 Fig. 16 is a flow chart showing the operation in 

photographing- storage mode 5 in the third embodiments- 
Fig. 17 is a table for explaining how a memory is 
used in the third embodiment; 

Fig. 18 is a block diagram showing the arrangement 
10 of a digital still camera; 

Fig. 19 is a block diagram showing an example of 
the arrangement of a digital camera according to the 
fourth embodiment of the present invention; 

Fig. 20 is a flow chart showing the photographing 
15 operation; 

Figs. 21A and 21B are flow charts for explaining 
object characteristic discrimination; 

Fig. 22 is a view for explaining a pixel shift 
method different from Fig. 7; 
20 Fig. 23 is a view for explaining the interpolation 

process corresponding to the pixel shift method shown in 
Fig. 22; 

Figs. 24A to 24D are views for explaining the pixel 
shift method shown in Fig. 7 in detail; and 
25 Figs. 25A to 25D are views for explaining the pixel 

shift method shown in Fig. 23 in detail. 
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DESCRIPTION OF THE PREFERRED EMBODIMENTS 
The preferred embodiments of an imaging apparatus 
and method, and its control method according to the 
5 present invention will be described in detail 

hereinafter with reference to the accompanying drawings. 
First Embodiment 
[Arrangement ] 

Fig. 1 is a block diagram showing the overall 
10 arrangement of a digital still camera (to be simply 

referred to as a camera hereinafter) in this embodiment. 
Referring to Fig. 1, reference numeral 1 denotes an 
overall controller which includes a microprocessor 
(containing a CPU, ROM, RAM, and the like) for 
15 controlling the overall camera. Reference numeral 2 
denotes a photographing mode setting unit for setting 
the photographing mode of the camera. The setting unit 2 
has switches and the like, which allow the user to set a 
"sports mode" suitable for photographing a moving 
20 object, a "portrait mode" suitable for photographing a 

still object, and the like. Reference numeral 18 denotes 
a camera operation unit for instructing the start of 
photographing by the camera and the like. The camera 
operation unit 18 comprises a main switch, release 
25 switch, and the like. 

Reference numeral 3 denotes a main photographing 
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optical system; and 4, an optical unit for spatially 
translating an object image formed on an imaging unit 6, 
as will be described later. The optical unit 4 uses a 
so-called shift correction optical system, as disclosed 
in Japanese Patent Laid-Open No. 3-186823. Normally, the 
shift correction optical system is used in a camera 
shake prevention mechanism for a camera. Reference 
numeral 17 denotes a vibration sensor which normally 
uses two angular velocity sensors called vibration 
gyros, and independently detects angular vibrations 
about two different axes, thus detecting the camera 
shake amount of the camera. In the camera of this 
embodiment, the overall controller 1 converts the output 
from the vibration sensor 17 into data for driving the 
optical unit 4, and moves lenses in the optical unit 4 
via a lens driver 5, thus always stably forming an 
object image on a predetermined imaging plane even by 
hand-held photographing . 

An object image signal formed on the imaging unit 6 
is converted into predetermined digital data via a 
series of video signal processes to be described below 
in the blocks before a monitor display 16 under the 
control of the overall controller 1. 
[Operation] 

Charges corresponding to an object image, which 
have been accumulated for a predetermined period of time 
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by the imaging unit 6 comprising, e.g., a CCD, are read 
out in turn, and luminance information of the object 
corresponding to the charge amount is converted into 
digital data by an A/D converter 7. Note that an optical 
color filter for separating photographed light into 
three primary colors R, G, and B, or the like is adhered 
onto the imaging unit 6. Hence, the imaging unit 6 
outputs signals indicating R, G, and B colors. 

The output from the A/D converter 7 is input to a 
process controller 8, and is subjected to dark level 
correction, gamma conversion, and the like. After that, 
the processed data is input to an image processor 9. 

The image processing in the image processor 9 will 
be explained below with reference to Fig. 2. The image 
processor 9 generates RGB data in units of pixels using 
different interpolation filters for green (G) and for 
red (R) and blue (B) with respect to the optical color 
filter array of the imaging unit 6, as shown in Fig. 2. 
Note that the optical color filter has a normal Bayer 
pattern, the green interpolation filter has a 
checkerboard pattern, and the interpolation filter for 
red and blue has a line sequential pattern, which are 
different from each other. In general, when the imaging 
unit 6 comprises a single imaging device, the imaging 
device has a Bayer pattern, as depicted on the left side 
in Fig. 2. Hence, since all pixels of image data 
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obtained from the imaging device do not always have all 
of RGB color information, the image processor 9 
generates RGB color information at all pixel points on 
the imaging device by interpolation filter calculations 

5 using 3x3 matrices depicted at the center of Fig. 2. 

In the pixel array of the imaging device depicted 
on the left side in Fig. 2, green data on pixel a can be 
obtained by respectively multiplying Ixaminance data of 
pixel a and its eight surrounding pixels by coefficients 

0 of the green interpolation filter. The coefficient of 
the green interpolation filter corresponding to the 
position of pixel a is "1", those of the upper, lower, 
and right, and left neighboring pixels are "0.25", and 
those of the obliquely neighboring pixels are "0". 

5 Therefore, in the green data on pixel a, the output 
value of pixel a is dominant. 

On the other hand, green data on pixel b is 
obtained by respectively multiplying luminance data of 
pixel b and its eight surrounding pixels by coefficients 

0 of the green interpolation filter. In this case, since 
the green output value of pixel b that outputs blue data 
is "0", the green data on pixel b is determined by 
averaging the green output values of the four, upper, 
lower, right, and left neighboring pixels. 

3 In the red and blue interpolation filter 

calculations as well, red and blue data at all pixel - 
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points are determined by substantially the same sequence 
except for their filter coefficients, which are 
different from those of the green interpolation filter. 
In this fashion, R, G, and B color output data for all 
the pixel points can be finally generated, as depicted 
on the right side in Fig. 2. 

The RGB data calculated in this way by the image 
processor 9 are transferred to a video memory 15 in 
units of frames to make finder display on the monitor 
display 16. 

The operation including the image processing before 
photographing has been explained. The operation 
including image processing upon photographing will be 
explained below. 

Upon photographing, the output from the process 
controller 8 is transferred to a frame memory unit 11 
having J frame memories 1 to J, and all pixel data are 
temporarily stored. The image processor 9 then processes 
the image data stored in this frame memory unit 11 by 
the above-mentioned method, and RGB data of the 
respective pixels output from the image processor 9 are 
transferred to a work memory 13. Furthermore, a memory 
controller 10 compresses the image data stored in the 
work memory 13 in a predetermined compression format, 
and saves the compressed data in an external memory 14 
comprising a nonvolatile memory such as a flash memory 
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or the like. Note that the external memory 14 may be a 
SMART medium which uses an EEPROM, a CF card that uses a 
flash memory, and the like. Also, magnetic storage media 
such as a floppy disk (FD) , magneto optical (MO) disk, 
hard disk (HD) mounted on a PC card, and the like can be 
used. 

A case will be explained below wherein the user 
obseirves an already photographed image. 

Image data saved in the external memory is 
temporarily transferred to the memory controller 10, and 
is expanded there in correspondence with the 
predetermined compression format. The expanded data is 
transferred to the work memory 13. Furthermore, the 
image data on the work memory 13 is transferred to the 
video memory 15 via the image processor 9. In this way, 
the already photographed image can be displayed on the 
monitor display 16. 
[Photographing Sequence] 

The photographing sequence in the camera of this 
embodiment will be briefly described below with 
reference to the flow chart in Fig. 3. Note that the 
photographing sequence to be described below is 
controlled by the overall controller 1. 

In the first step SlOO, it is checked if the main 
switch in the camera operation unit 18 is ON. If the 
user has turned on the main switch, the flow immediately 
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advances to step SI 01 to supply electric power to all 
the blocks of the camera shown in Fig. 1. Subsequently, 
the vibration sensor 17 is enabled in step SI 02 to begin 
to detect the camera shake amount. In step S103, the 
optical unit 4 is driven via the lens driver 5, as 
described above, thus starting camera shake correction. 

In step S104, the output from the imaging unit 6 is 
converted into a video signal via the A/D converter 7, 
process controller 8, and image processor 9, as 
described above. In step S105, a monitor display 
corresponding to the video signal is started. Hence, in 
step S105 and the subsequent steps, the above-mentioned 
video signal processing repeats itself in units of 
frames of the monitor display. In step 8106, whether or 
not the release switch in the camera operation unit 18 
is ON is detected to check if the user has released the 
shutter button. If the release switch is ON, 
photographing is immediately started. 

Upon starting photographing, the photographing mode 
setup of the camera is checked in step S107 . In the 
camera of this embodiment, the photographing mode 
setting unit 2 allows the user to set a so-called 
"sports mode" for mainly photographing a moving object, 
or a so-called "portrait mode" for mainly photographing 
a still object. If the "sports mode" is set, the flow 
advances to step S108 to set a "normal photographing 



- 18 - 



mode" as a mode upon photographing. In step S109, 
photographing and storage of photographed data in the 
frame memoiry unit 11 are made in "photographing* storage 
mode 1". The operation in this "photographing- storage 
mode 1" will be described later. 

On the other hand, if the "portrait mode" is set, 
the flow advances to step SllO to set a "pixel shift 
photographing mode" as a mode upon photographing. In 
step Sill, photographing and storage of photographed 
data in the frame memory unit 11 are made in 
"photographing- storage mode 2". The operation in this 
"photographing- storage mode 2" will be described later. 

In this fashion, after photographed data is stored 
in the frame memory unit 11 in either mode, it is 
checked in step S112 if the release switch is OFF. If 
the release switch is kept ON, the processing stays in 
step S112; otherwise, the flow returns to step S106. 
Note that the sequence returns to step SlOO if the main 
switch is turned off, although not shown in Fig, 3, 

As described above, in this embodiment, 
photographing is automatically switched to "normal 
photographing" or "pixel shift photographing" in 
correspondence with the photographing mode such as the 
"sports mode" or "portrait mode" set by the user, and 
the saving method (including the compression method) of 
image data in the frame memory unit 11 is switched. 
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The operations in "photographing- storage mode 1" 
and "photographing- storage mode 2" above will be 
described in detail below. 
[Photographing- storage Mode 1] 

"Photographing- storage mode 1" used when the 
"sports mode" is set will be explained below with 
reference to the flow chart in Fig. 4. 

The output from the process controller 8 is 
temporarily stored in one of a plurality of frame 
memories in the frame memory unit 11. For this purpose, 
for example, "1" is set in parameter k for selecting a 
frame memory, thus designating frame memory 1, in step 
S200. In step S201, the control waits for completion of 
charge accumulation in the imaging unit 6. When the 
imaging unit 6 comprises, e.g., a CCD or the like, 
charges generated by photoelectric conversion are 
immediately transferred to a transfer unit upon 
completion of accumulation for a predetermined period of 
time. Hence, even while the generated charges are read 
out in turn, the next accumulation has already been 
started. 

In step S2 02, the above-mentioned process control 
is done in imits of pixel data in the process controller 
8, and the processing results are sequentially stored in 
frame memory k (frame memory 1 in this case) . If it is 
determined in step S203 that all pixel data for one 
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frame have been stored in frame memory k, the flow 
advances to step S204. 

In step S204, the storage contents of frame memory 
k are transferred to the image processor 9 to generate 
5 RGB data in units of pixels by the interpolation 
described above. In step S205, the RGB data are 
temporarily transferred to the work memory 13 . This 
processing continues for one frame, and if it is 
determined in step S206 that processing for one frame is 

10 complete, the flow advances to step S207. 

In steps S207 to S211, an actual photographed image 
is compressed and its data is saved. In step S207, 
lossless compression is set as the image compression 
method in the memory controller 10. As the lossless 

15 compression, DPCM (Differential Pulse Code Modulation) 
or the like is used, DPCM modulates and encodes only 
differences between neighboring pixels of image data. 
According to this method, a compression rate (data size 
of compressed image/data size of original image x 100%) 

20 with respect to the original image is as low as about 
50%, but encoded photographed data can be completely 
decoded to reclaim the original image independently of 
the type of object. Hence, DPCM is suitably used when 
the user does not want any drop of image quality due to 

25 image compression. 

Fig. 6 shows an example of compression by DPCM, and 
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the compression process will be explained below. 
Assuming that an original image has a two-dimensional 
array of luminance signals, (or R, G, and B color 
signals), as depicted by the upper matrix in Fig. 6, the 
array of the original image is converted into a linear 
array, as shown in the lower table in Fig. 6. This 
conversion is implemented by tracing pixels of the 
original image in turn from the upper left to the right 
and, upon reaching the right-end pixel, tracing pixels 
from the one immediately below the right -end pixel to 
the left. In this manner, the pixel data of the original 
image are sequentially converted into a linear array, 
and the differences between neighboring pixel data on 
this linear array are encoded, thus compressing the 
data . 

Hence, lossless compression such as DPCM is 
executed in units of predetermined blocks of an original 
image in step S208 in Fig. 4, and the compressed image 
data is converted into code data using, e.g. Huffman 
coding or the like in step S209. Note that the lossless 
compression need not always be done in units of blocks. 
Huffman coding assigns a larger code length to data with 
lower probability of occurrence to encode data. In step 
S210, code data as the encoded image data is 
sequentially stored in the external memory 14. If it is 
determined in step S211 that compression of the entire 
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image (all blocks) and saving of the compressed data in 
the external memory 14 are complete, the processing in 
"photographing- storage mode 1" ends. Of course, the 
processing in steps S208 to S211 repeats until 
cort^ression of the entire image (all blocks) and saving 
of the compressed data in the external memory 14 are 
completed. 

[Pixel Shift Photographing] 

Prior to the description of "photographing- storage 
mode 2" used when the "portrait mode" is set, pixel 
shift photographing as the photographing method of 
"photographing- storage mode 2" will be described with 
reference to Fig. 7. 

The upper matrix in Fig, 7 depicts the RGB array of 
an initial image 701, which has a Bayer pattern, as 
described above. When the optical unit 4 is decentered 
by a predetermined amount in the X-direction in Fig. 7 
during the next one frame period, an image 702, which 
has been shifted by one pixel pitch P from the initial 
image 701 in the horizontal direction, as indicated by 
the lower left matrix in Fig. 7, can be obtained. This 
process will be referred to as "the first shift" 
hereinafter. With this single shift, the spatial 
frequency of an image can be doubled in the X-direction 
in principle. 

Subsequently, by decentering the optical unit 4 in 
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the first shift state in the -X- and Y-directions by a 
predetermined amount, an image 703, which has been 
shifted from the first shift image 702 by -P/2 in the 
X-direction and P/2 in the Y-direction, as indicated by 
the lower central matrix in Fig. 7, can be obtained. 
This process will be referred to as "the second shift" 
hereinafter. Furthermore, by decentering the optical 
unit 4 in the second shift state in the -X-direction 
again by a predetermined amount, an image 704, which has 
been shifted from the second shift image 703 by P in the 
X-direction, as indicated by the lower right matrix in 
Fig. 7, can be obtained. This process will be referred 
to as "the third shift" hereinafter. 

In this way, when the initial image 701, and the 
first, second, and third shift images 702, 703, and 704, 
which are shifted by predetermined pixel pitches, are 
photographed, the spatial frequency of the image can be 
nearly doubled in both the X- and Y-directions by 
combining these four photographed image data. 

Note that the operation called "pixel shift" in 
this embodiment is to shift the imaging position on the 
imaging plane, as can be seen from the above 
description. That is, "pixel shift" does not actually 
shift pixels of the imaging device, but a shift of the 
imaging position can be considered equivalent to a pixel 
shift. Hence, in this specification, the operation for 
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shifting the imaging position on the imaging plane will 
be referred to as "pixel shift", and photographing with 
"pixel shift" will be referred to as "pixel shift 
photographing" . 
[Photographing- storage Mode 2] 

The operation in "photographing- storage mode 2" 
with pixel shift photographing will be esqplained below 
with reference to the flow charts in Figs. 5A and 5B. 

In step S250, "1" is set in parameter k for 
selecting a frame memory for storing the output from the 
process controller 8, thus designating frame memory 1. 
In step S251, the control waits for completion of charge 
accumulation in the imaging unit 6. 

Upon completion of charge accumulation 
corresponding to the initial image 701 shown in Fig. 7 
in the imaging unit 6, drive amounts AX(K) and AY(K) for 
decentering the optical unit 4 are set in steps S252 and 
S253 to attain the first shift, and the optical unit 4 
is decentered via the lens driver 5. In this case, a 
value for shifting the object image from the initial 
image 701 by one pixel pitch P on the imaging plane is 
set in the first drive amount AX(1), and zero is set in 
AY(1) not to decenter the optical unit 4 in the 
Y-direction. 

In step S254, the process controller 8 executes the 
aforementioned process control of individual pixel data. 
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and the processing results are sequentially stored in 
frame memory K (frame memory 1 in this case) . If it is 
determined in step S255 that image data for one frame 
corresponding to the initial image 701 has been stored 
in frame memory 1, the flow advances to step S256. 

It is checked in step S256 if the value of 
parameter K is N ("4" in this embodiment). If NO in step 
S256, K is incremented by 1 in step S257, and it is 
checked again in step S251 if charge accumulation for 
the next one frame is complete. Upon completion of 
charge accumulation, a value for shifting the image by 
half a pixel pitch -P/2 is set in AX(2) and a value for 
shifting the image by half a pixel pitch P/2 is set in 
AY(2) in steps S252 and S253 . After that, the 
operations in steps S254 to S257 repeat themselves. At 
that time, frame memory 2 stores an image corresponding 
to the image 702 shown in Fig. 7. 

Furthermore, upon executing steps S252 and S253 
when K = 3, a value for shifting the image by one pixel 
pitch -P is set in AX(3) and zero is set in AY (3) in 
turn. On the other hand, upon executing steps S252 and 
S253 when K = 4, P/2 is set in AX(4) and -P/2 is set in 
AY (4) to set zero decentering amount of the optical unit 
4. 

In this way, by repeating steps S251 to S257 until 
the value K becomes equal to N ("4" in this embodiment) 
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in step S256, images for four frames, which correspond 
to the images 701 to 704 in Fig. 7 and have been shifted 
by one or a half pixel pitch in the X-direction and/or 
Y-direction, are stored in frame memories 1 to 4 . 

Fig. 8 shows the movement of the optical unit 4 
upon pixel shift photographing, and the movement will be 
described in detail below. Fig. 8 illustrates the 
movements of an object image in the X- and Y-directions 
by the optical unit 4 in relation to a time axis t. 

Since camera shake correction is being made before 
photographing, the optical unit 4 is moving the object 
image in the X- and Y-directions in correspondence with 
the output from the vibration sensor 17 even before an 
effective photographing period shown in Fig. 8. 

During the effective photographing period, the 
optical unit 4 is decentered by an amount corresponding 
to AX(1) in only the X-direction after completion of the 
first charge accumulation, and the second charge 
accumulation starts while continuing camera shake 
correction in that state. After completion of the second 
charge accumulation, the optical unit 4 is decentered by 
amounts corresponding to AX(2) and AY (2) in the X- and 
Y-directions, and the third charge accumulation then 
starts. Upon completion of the third charge 
accumulation, the optical unit 4 is decentered by an 
amount corresponding to AX(3) in only the X-direction. 
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Upon completion of the fourth charge accumulation, the 
decentered amounts in the X- and Y-directions are reset 
to complete pixel shift photographing. 

In this fashion, images for four frames are 
obtained by pixel shift photographing. Subsequently, in 
step S258 and the subsequent steps, high-density image 
data obtained by pixel shift photographing is converted 
into RGB information. 

In step S258, the value of parameter K is set at 
"1" to designate the frame memory which stores the image 
data obtained by the first pixel shift photographing. 
Subsequently, the image data stored in frame memory 1 is 
transferred to the image processor 9 (S259) . In this 
case, unlike in "photographing- storage mode 1" mentioned 
above, no interpolation is executed, and it is checked 
in step S260 if transfer for one frame is complete. If 
it is determined in step S260 that transfer for one 
frame is complete, it is checked in step S261 if the 
value K is equal to N ("4" in this embodiment), so as to 
check if all the photographed image data have been 
transferred. If all the photographed image data have not 
been transferred yet, K is incremented by "1" in step 
S262, and the flow returns to step S259 to start 
transfer of image data stored in the next frame memory. 

Upon completion of transfer of all the photographed 
data, the flow advances to step 8263 to interpolate 
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using all the photographed image data. 

The interpolation process will be explained below 
with reference to Fig. 9. The left matrix in Fig. 9 
shows the array of pixel data obtained by pixel shift 
5 photographing, which has a spatial frequency nearly 

twice that of the imaging elements in the Bayer pattern 
shown in Fig. 2 in both the X- and Y-directions . In 
order to obtain RGB data doubled in both X- and 
Y-directions from that pixel data array, filter 

10 calculations using interpolation filters configured by 
central matrices shown in Fig. 9 are made. 

As for the green component, the upper central 3x3 
matrix in Fig. 9, which is the same as that in Fig. 2, 
suffices. For example, when green data at the position 

15 of pixel a in the left pixel data array in Fig. 9 is to 
be generated, luminance data of pixel a and its eight 
surrounding pixels bounded by the dotted line need only 
be multiplied by the corresponding coefficient of the 
green interpolation filter. In this case, since the 

20 coefficient corresponding to the green output at the 

position of pixel a is "1", those of the upper, lower, 
and right, and left neighboring pixels are "0.25", and 
those of the obliquely neighboring pixels are "0", the 
output value of pixel a is dominant in the green data on 

25 pixel a. 

On the other hand, in order to generate green data 
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of a pixel corresponding to position b in the left pixel 
data array shown in Fig. 9, liuninance data of the eight 
pixels around position b bounded by the dotted line need 
only be multiplied by the corresponding coefficients of 
the green interpolation filter. However, since there is 
no green data at position b, green data corresponding to 
that position is generated by the average value of the 
upper, lower, right, and left neighboring green data. 

As for the red and blue data, more complex 
processing than green data is required. As can be seen 
from the left pixel data array shown in Fig. 9, red and 
blue data can be interpolated using immediately 
neighboring pixel data in the X-direction, but must be 
interpolated using pixel data at slightly separated 
positions in the Y-direction. Hence, the lower central 5 
X 5 matrix in Fig. 9 is used. Note that in this matrix 
the coefficient columns having the matrix center as the 
center are not the same as the coefficient rows. When 
such filter calculations are done for the entire pixel 
data array, RGB data corresponding to the right pixel 
data array shown in Fig. 9 is finally obtained. 

Upon completing interpolation of image data for 
four frames obtained by pixel shift photographing in the 
image processor 9, image data is temporarily transferred 
to the work memory 13 in step S264 to compress and save 
the image data obtained by interpolation. Subsequently, 
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lossy compression is set as the image compression method 
in the memory controller 10 in step S265. 

Note that in lossy compression data exactly the 
same as that before compression cannot be reconstructed 
from compressed data. If lossy compression is done using 
DCT (Discrete Cosine Transform) to transforiTi and 
quantize image data in the respective blocks into 
two-dimensional frequency data, the image data volume 
can be greatly reduced. 

Fig. 10 illustrates the compression process using 
DCT. As mentioned above, in JPEG compression or the like 
using DCT, the entire image is normally divided into 8 x 
8 blocks, and similar transformation is done for each 
block. The left matrix in Fig. 10 exemplifies this 8x8 
pixel block, and the upper central DCT coefficient block 
in Fig. 10 represents the DCT given below for the 
respective pixel data: 

Sij = iciCjxi:[(Pxy-128)cos{2x+l)i^}cos{(2y+I)jiL}] 

^ i=Oj=0 16 16 

...{!) 

for Ci, Cj = I/V2 when i, j = 0 
Ci, Cj = 1 when i, 3^1 

The upper left coefficient in the upper central DCT 
coefficient block in Fig. 10 is the direct current (DC) 
component contained in the image, and other coefficients 
are alternate current (AC) components which indicate the 



levels of high-frequency components contained in the 
image toward the lower right. The respective 
coefficients included in the DCT coefficient block are 
quantized by the lower central quantization table in 
5 Fig. 10 to obtain the right quantization coefficient 

block in Fig. 10. The respective coefficients included 
in the quantization coefficient block are finally 
encoded by Huffman coding. 

Note that quantization transforms coefficient s in 

10 the DCT coefficient block into a quotient obtained by 

dividing it by data at the corresponding position in the 
quantization table. For example, in the DCT coefficient 
block, the value of pixel S^^ at i = j = 0 is "260", and 
the quantization result of this pixel value using data 

15 "16" at the corresponding position in the quantization 

table is "16", which appears at the upper left corner of 
the quantization coefficient block. On the other hand, 
the value of pixel S30 at i = 3 and j = 0 is "5", and the 
quantization result of this pixel value using data "16" 

20 at the corresponding position in the quantization table 
is "0". Hence, by changing the individual data in the 
quantization table with respect to the DCT coefficient 
block as the conversion result into frequency data, the 
value of the coefficient to be encoded can be controlled 

25 to decrease, and the number of "0" quantization 

coefficients can also be controlled, thus arbitrarily 
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setting the image compression level. 

Therefore, lossy compression using DCT is executed 
in units of 8 x 8 pixel blocks for the interpolated 
image after pixel shift photographing in step S266, and 
the lossy-compressed image data blocks are encoded by, 
e.g., Huffman coding in step S267 as in 

"photographing- storage mode 1" above. The encoded image 
data blocks are sequentially stored in the external 
memory 14 in step S268. If it is determined in step S269 
that compression of the entire image (all the blocks) 
and storage of it to the external memory 14 are 
complete, the processing in "photographing- storage mode 
2" ends. 

In the above description, a high-resolution image 
obtained by pixel shift photographing is saved in the 
memory after lossy compression. Application of the 
compression method that admittedly deteriorates image 
quality to a high-resolution image may seem 
contradictory. However, image deterioration due to image 
compression is negligible unless an extremely high 
compression rate is set. Hence, by obtaining a 
high-resolution image by pixel shift photographing and 
setting a compression rate within an allowable range of 
image quality deterioration due to compression, high 
image quality can be maintained. 

Note that this embodiment has exemplified a case 
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wherein images for four frames are taken by pixel shift 
photographing, and are synthesized by interpolation to 
obtain a single photographed image. Of course, the 
number of frames to be taken by pixel shift 
photographing is not limited to such specific example, 
but can be increased to its upper limit which may occur 
in consideration of the time required for photographing, 
the number of available frame memories, the time 
required for interpolation, and the memory capacity. 

As described above, according to this embodiment, 
since the camera itself automatically determines normal 
or pixel shift photographing to be done in 
correspondence with the photographing mode set by the 
user, pictures with image quality meeting the user's 
requirement can be taken. Furthermore, since the camera 
automatically determines an optimal compression method 
upon saving photographed images in the memory, the user 
need not determine any compression method and the like. 
Hence, even a user who is not skilled in digital images 
can appropriately take a picture using a digital still 
camera by only selecting the photographing mode. 
Second Embodiment 

The second embodiment according to the present 
invention will be described below with reference to the 
flow charts in Figs. 11 to 14. Since the arrangement of 
a digital still camera in the second embodiment is the 
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same as that in Fig. 1, the same reference numerals in 
the second embodiment denote the same parts as those in 
the first embodiment, and a detailed description thereof 
will be omitted. 

In the first embodiment described above, the 
photographing method and the saving method (compression 
method) of photographed image data are determined in 
correspondence with the photographing mode set by the 
user. In the second embodiment, the photographing method 
and the saving method (compression method) of 
photographed image data are determined in correspondence 
with the vibration state upon actual photographing. 

Fig, 11 is a flow chart showing the photographing 
sequence in the camera of the second embodiment. Note 
that this photographing sequence is controlled by the 
overall controller 1 of the camera • 

Since photographing preparation and photographing 
mode setup processes of the camera in steps SlOO to 
S108, SllO, and S112 have already been described in the 
first embodiment with the aid of Fig. 3, a detailed 
description thereof will be omitted. 

If the "normal photographing mode" is set, the flow 
advances to step S3 09 to check if a "continuous 
photographing mode" is set. In the "continuous 
photographing mode", a plurality of images are 
continuously taken by single release operation of the 
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camera, and the results are saved in the memory, if it 
is determined in step S3 09 that the "continuous 
photographing mode" is not set, the flow advances to 
step S3 10 to actually take a picture in 
5 "photographing, storage mode 1" and to store photographed 
data in the frame memory unit 11, The operation in this 
"photographing- storage mode 1" is the same as the 
processing shown in the flow chart in Fig. 4 in the 
first embodiment mentioned above, and a detailed 

10 description thereof will be omitted. But to reiterate, 
standard photographing and storage for converting a 
photographed image into a lossless-compressed file and 
saving the converted file in the memory are done. 

On the other hand, if it is determined in step S3 09 

15 that the "continuous photographing mode" is set, the 
flow advances to step S311 to execute 
"photographing- storage mode 3". In this 
"photographing- storage mode 3", only an image of the 
first frame taken by continuous photographing is 

20 lossless-compressed, and images of other frames are 

lossy-compressed to minimize the data size to be saved 
upon continuous photographing. This mode will be 
described in detail later. 

If the "pixel shift photographing mode" is set, it 

25 is checked in step S3 13 if the current vibration amount 
is large. The vibration amount is determined by 
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referring to the output from the vibration sensor 17 or 
detecting vibration produced on the imaging plane on the 
basis of image correlation in units of frames in the 
imaging unit 6. When the vibration amount is determined 
based on image correlation in units of frames, the 
vibration amount that remains after vibration correction 
by the optical unit 4 can be accurately detected. A 
detailed description of this determination method will 
be omitted here. 

If it is deteirmined in step S3 13 that the vibration 
amount is smaller than a predetermined amount, the flow 
advances to step S3 14 to execute "photographing- storage 
mode 2". Since "photographing- storage mode 2" is the 
same as that in the first embodiment, a detailed 
description thereof will be omitted. Briefly stated, a 
high-resolution image is obtained by a plurality of 
number of times of pixel shift photographing, and 
lossy-compressed image data is saved. 

On the other hand, if the vibration amount is 
larger than the predetermined amount, it is improper to 
use "photographing- storage mode 2". This is because 
since the spatial position of the imaging unit is 
shifted along with time in pixel shift photographing, 
image data for a plurality of frames are required. 
Hence, pixel shift photographing is readily influenced 
by vibrations since the photographing time is long. More 
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specifically, the shift amount resulting from vibrations 
becomes larger than that for one or a half pixel by 
pixel shift, and the spatial array relationship among 
the individual images obtained by pixel shift 
photographing cannot be maintained upon interpolation 
after pixel shift photographing. As a result, the image 
quality of an image obtained by interpolation 
deteriorates considerably. 

Hence, if the vibration amount is larger than the 
predetermined amount, the flow advances to step S3 15 to 
execute "photographing- storage mode 4". In 
"photographing- storage mode 4", even when the "pixel 
shift photographing mode" is set, since pixel shift 
cannot improve image quality, standard photographing and 
storage of image data like in "photographing- storage 
mode 1" are done. This mode will be explained in detail 
later. 

The operations in "photographing- storage mode 3" 
and "photographing- storage mode 4" will be explained in 

detail below. 

[Photographing- storage Mode 3] 

The operation in "photographing- storage mode 3" 
upon setting the "continuous mode" will be explained 
below with reference to the flow charts in Figs. 12 and 
13. 

In step S3 50, "1" is set in parameter K for 
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selecting the frame memory that temporarily stores the 
output from the process controller 8, thus designating 
frame memory 1. In step S351, the control waits for 
completion of charge accumulation in the imaging unit 6. 
As has been described in the first embodiment as well, 
when the imaging unit 6 comprises a CCD or the like, 
since charges produced by photoelectric conversion are 
transferred to a transfer unit immediately upon 
completion of accumulation for a predetermined period of 
time, the next charge accumulation has been started even 
while the produced charges are read out in turn. 

Hence, the results of the aforementioned process 
control in units of pixels are sequentially stored in 
frame memory K (in this case, frame memory 1) in step 
S352, and if it is detected in step S353 that all pixel 
data in one frame have been stored in frame memory K, 
the flow advances to step S354. 

It is checked in step S354 if the value of 
parameter K that designates a frame memory is equal to 
M, so as to check if photographing has repeated itself M 
times in the "continuous photographing mode". Of course, 
the number M of times of photographing is set in advance 
by the user as the number of pictures to be continuously 
taken. If the value K has not reached M yet, the flow 
advances to step S355 to increment K by "1", and the 
flow returns to step S351 again to check if the next 



charge accumulation is complete. Upon completion of 
photographing M times, it is determined in step S354 
that the value of parameter K is equal to M, and the 
flow advances to step S356. 

In step S3 56, "1" is set in parameter K to 
designate a frame memory. Subsequently, in step S357, 
the storage contents of frame memory K (frame memory 1 
in this case) are transferred to the image processor 9, 
and are subjected to interpolation, which has been 
described in the first embodiment. The interpolation 
result is temporarily transferred to the work memory 13 
in step S358. This operation repeats itself for one 
frame, and if it is detected in step S3 59 that the 
processing for one frame is complete, the flow advances 
to step S360. 

In step 83 60, lossless compression is set in the 
memory controller 10 as the image compression method. 
For example, the lossless compression uses DPCM 
described in "photographing- storage mode 1" described in 
the first embodiment. Since the processing in steps S3 61 
to S364 is the same as that in steps S208 to S211 in 
Fig. 4 that shows the operation in 

"photographing- storage mode 1", a detailed description 
thereof will be omitted. With this aforementioned 
processing, an image of the first frame taken by 
continuous photographing is lossless-compressed and 
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saved . 

Upon completing compression and saving of all 
blocks of the image of the first frame, the flow 
advances to step S365 to set "2" in parameter K so as to 
select the next frame memory. Subsequently, the storage 
contents of frame memory K (frame memory 2 in this case) 
are transferred to the image processor 9 in step S366, 
and the interpolated image is transferred to the work 
memory 13 in step S3 67. This operation continues for one 
frame, and if it is detected in step S3 68 that the 
processing for one frame is complete, the flow advances 
to step S3 69. 

In step S369, lossy compression is set in the 
memory controller 10 as the image compression method. 
For example, the lossy compression uses compression 
using DCT, which has been described in 

"photographing- storage mode 2" in the first embodiment. 
Hence, DCT and quantization are executed after the 
photographed image is broken up into, e.g., 8x8 pixel 
blocks in step S370, and the quantization results are 
encoded by, e.g., Huffman coding in step S371. Then, the 
code data are saved in the external memory 14 in step 
S372. Furthermore, it is checked in step S373 if all 
blocks have been compressed and saved. If all blocks 
have not been compressed and saved yet, the flow returns 
to step S370 to repeat the above operations. Upon 
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completion of compression and storage of all the blocks 
in the external memory 14, the flow advances to step 
S374. 

It is checked in step S374 if the value of 
parameter K for selecting a frame memory has reached the 
number M of times of photographing, so as to check if 
compression and saving of all photographed images are 
complete. If compression and saving are not complete 
yet, the value K is incremented by "1" in step S375, and 
the flow returns to step S366 to repeat compression and 
saving of the next image. Hence, upon completion of 
compression and saving for M frames of images, the 
operation in "photographing- storage mode 3" ends. 

As mentioned above, in "photographing- storage mode 
3", only the image of the first frame taken by 
continuous photographing is lossless -compressed, and 
images of other frames are lossy-compressed. That is, 
data compression and saving dedicated to the continuous 
photographing mode are done. 
[Photographing- storage Mode 4] 

The operation in "photographing- storage mode 4" 
selected when the vibration amount is large will be 
explained below with reference to the flow charts in 
Figs. 14A and 14B. 

The operation in steps S400 to S407 takes all 
images by pixel shift photographing and stores them in 
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the frame memories. Since this operation is the same as 
that in steps S250 to S257 in the flow charts of 
Figs. 5A and 5B that show "photographing- storage mode 2" 
in the first embodiment above, a detailed description 
thereof will be omitted. 

Upon completion of photographing of a predetermined 
niamber of images by pixel shift photographing, "1" is 
set in parameter K for designating a frame memory in 
step S408. Subsequently, the storage contents of frame 
memory K (frame memory 1 in this case) are transferred 
to the image processor 9 and are subjected to 
interpolation mentioned above in step S409 . The 
interpolation result is temporarily transferred to the 
work memory 14 in step S410. This operation continues 
for one frame, and if it is detected in step S411 that 
the processing for one frame is complete, the flow 
advances to step S412. 

In step S412, lossless compression is set in the 
memory controller 10 as the image compression method. 
For example, the lossless compression uses DPCM 
described in "photographing- storage mode 1" in the first 
embodiment. Since the processing in steps S413 to S416 
is the same as that in steps S208 to S211 in 
"photographing- storage mode 1" shown in Fig. 4, a 
detailed description thereof will be omitted. 

If it is determined in step S416 that compression 
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and saving of all blocks of an image of the first frame 
are complete, the flow advances to step S417 to check if 
compression and saving of all images taken by pixel 
shift photographing are complete. If compression and 
saving of all images are not yet complete, the flow 
advances to step S418 to increment the value of 
parameter K for selecting a frame memory. The flow then 
returns to step S409 to compress and save the next 
image. More specifically, N photographed image data 
taken by pixel shift photographing are 

lossless-compressed and saved in the external memory 14. 

As described above, in the second embodiment, even 
when the pixel shift photographing mode is automatically 
selected, if the vibration amount before photographing 
is large and improvement of image quality cannot be 
expected by pixel shift photographing, not only 
interpolation for synthesizing a plurality of images 
taken by pixel shift photographing is disabled, but also 
the compression method is switched from lossy 
compression to lossless compression. That is, in 
correspondence with the vibration amount, 
"photographing- storage mode 4" for individually 
lossless-compressing and storing images taken by pixel 
shift photographing, or "photographing- storage mode 2" 
for lossy-compressing and storing one image obtained by 
synthesizing images taken by pixel shift photographing 
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is determined. 

In the above description, the vibration amount is 
detected before photographing, but may be detected 
during photographing. For example, the vibration amount 
detection step may be inserted among steps S400 to S407 
in Fig. 14A. 

In the above description, when the detected 
vibration amount is large, a plurality of images taken 
by pixel shift photographing are not synthesized even 
when the pixel shift photographing mode is set. 
Alternatively, for example, when the vibration amount is 
large, pixel shift photographing itself may be disabled. 

To restate, according to the second embodiment, 
when the camera shake amount is large upon 
photographing, even when the pixel shift photographing 
mode is set, a plurality of images taken by pixel shift 
photographing are individually saved without being 
synthesized, and the compression method upon saving is 
also automatically switched from lossy compression to 
lossless compression, thus minimizing deterioration of 
image quality arising from camera shake. 
Third Embodiment 

The third embodiment according to the present 
invention will be described below with reference to the 
flow charts in Figs. 15A, 15B, and 16. Since the 
arrangement of a digital still camera in the third 
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embodiment is the same as that in Fig. 1, the same 
reference numerals in the third embodiment denote the 
same as those in Fig. 1, and a detailed description 
thereof will be omitted. 

In the first and second embodiments above, the 
photographing method and image data saving method, i.e., 
compression method, are determined in correspondence 
with the photographing mode and vibration amount. In the 
third embodiment, the photographing method and image 
data saving method (compression method) are determined 
in correspondence with the free space of the storage 
capacity of the external memory 14 that stores image 
data. 

Figs. 15A and 15B are flow charts showing the 
photographing sequence in the camera of the third 
embodiment. This photographing sequence is controlled by 
the overall controller 1 in the camera. Note that the 
photographing sequence shown in Figs. 15A and 15B is 
substantially the same as that in the second embodiment 
shown in Fig. 11, except that steps S510 to S513 of 
checking the free space of the external memory 14, and 
step S520 of executing "photographing- storage mode 5" 
are added. Hence, since the operation and processing in 
other steps shown in Figs. 15A and 15B have already been 
described above with the aid of Fig. 11, a detailed 
description thereof will be omitted. 



In the third embodiment, the flow advances to one 
of steps S510 to S513 in correspondence with the 
photographing mode set to check if the remaining storage 
capacity of the external memory 14 is sufficient. 

A method of checking if the remaining storage 
capacity of the external memory 14 is sufficient will be 
explained below with reference to Fig. 17. Fig. 17 shows 
the use state of the memory when photographed images are 
stored in the external memoory 14. After the external 
memory 14 is initialized, image data are stored in turn 
from lower addresses, i.e., image data taken by the 
first photographing is saved at address 0, image data 
taken by the second photographing is saved at address 1, 
and so forth. In this case, since the image data size 
varies in units of shots, the address value does not 
always match the image photographing number. However, 
the remaining memory capacity can be detected on the 
basis of end address m of the memory itself and the 
current address n+1 indicated by an address pointer. 
Note that the threshold values must be set in units of 
photographing modes in advance to check if the obtained 
remaining capacity is sufficient in the subsequent . 
processing. Of course, the threshold values may be 
changed by the user. 

If the remaining capacity of the external memory 14 
is sufficient, photographing and storage of image data 
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are done in the corresponding photographing- storage 
mode. 

On the other hand, if the remaining capacity of the 
external memory 14 is insufficient, the flow advances to 
step S520 to execute "photographing- storage mode 5". As 
will be described in detail later, in this 
"photographing- storage mode 5", the number of frames to 
be taken is limited to one (to be referred to "one-shot 
photographing" hereinafter) , that photographed image is 
compressed at a high compression rate, and the 
compressed data is stored in the external memory 14 so 
as to store a photographed image even when the remaining 
memory capacity is small. 

As described above, in the third embodiment, when 
the remaining capacity of the external memory 14 that 
stores photographed image data is insufficient for the 
set photographing- storage mode, "photographing- storage 
mode 5" for making one-shot photographing irrespective 
of the set photographing- storage mode, and compressing a 
photographed image at a high compression rate is 
executed. The operation in "photographing- storage mode 
5" will be described in detail below with reference to 
the flow chart in Fig. 16. 

Referring to Fig. 16, since the operation in steps 
S550 to S556 is the same as that in steps S200 to S206 
in "photographing- storage mode 1" shown in Fig. 4, a 
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detailed description thereof will be omitted. 

In step S557, lossy compression is set in the image 
controller 10 as the image compression method. For 
example, the lossy compression uses the DCT described in 
"photographing- storage mode 2" of the first embodiment 
to compress data using a quantization table, which can 
obtain a sufficiently high compression rate. Since steps 
S558 to S561 are the same as steps S265 to S269 in 
Figs. 5A and 5B, a detailed description thereof will be 
omitted. 

In this way, in "photographing- storage mode 5", 
since the remaining capacity of the external memory 14 
is small, one-shot photographing is done forcibly, and 
image data is compressed and saved by lossy compression 
with a high compression rate. When 

"photographing- storage mode 5" is or was executed, a 
message indicating the small remaining capacity of the 
external memory 14 and/or a message indicating that 
"photographing- storage mode 5" is or was executed are/is 
preferably displayed on the camera operation unit 18 
and/or monitor display 16 to call for user's attention. 

As described above, according to the third 
embodiment, when the remaining capacity of the external 
memory 14 is small, one-shot photographing and 
compression with a high compression rate are forcibly 
done irrespective of the current camera photographing 
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mode so as to reduce the image data volume to be stored. 
Hence, even when the remaining capacity of the external 
memory 14 becomes small, photographing can be done by 
fully using that remaining capacity. 

According to the embodiments described above, the 
following effects can be obtained: 

(1) Since the photographing method and saving 
method (compression method) of photographed image data 
are automatically switched in accordance with the 
photographing mode corresponding to an object, which is 
set in the imaging apparatus, an optimal photographing 
method and saving method of the photographed image can 
be automatically selected in correspondence with the 
object without requiring any troublesome operation. 

(2) Since the photographing method and saving 
method (compression method) of photographed image data 
are automatically switched in accordance with the state 
(camera shake or the like) upon photographing of the 
imaging apparatus, the influence of camera shake or the 
like on the photographed image can be minimized. 

(3) Since the photographing method and saving 
method (compression method) of the photographed image 
data are automatically switched in accordance with the 
remaining capacity of the memory that saves the 
photographed image, photographing can be done by fully 
using the limited remaining capacity. 
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To recapitulate, since the imaging apparatus 
automatically determines an appropriate photographing 
method and saving method (compression method) of 
photographed image data, even a user who is not skilled 
in operations of the digital still camera can easily and 
appropriately take a picture by only setting a 
photographing mode corresponding to an object of 
interest . 

In the above embodiments, DPCM is used as lossless 
compression. For example, the Lempel~Ziv scheme may be 
used instead. Similarly, a compression method such as 
JPEG that uses DCT has been exemplified as lossy 
compression. Alternatively, a compression method using 
wavelet transform may be used. That is, a combination of 
a compression method that can completely reconstruct 
image data before compression even if the compression 
rate is low, and a compression method which can assure a 
high compression rate even if it cannot completely 
reconstruct image data before compression need only be 
used, and the compression rate of the compression method 
with a high compression rate is preferably controllable , 
Fourth Embodiment 
t Arrangement ] 

Fig. 19 is a block diagram showing an example of 
the arrangement of a digital camera according to the 
fourth embodiment of the present invention. Since the 
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arrangement of the fourth embodiment is substantially 
the same as that in the first embodiment shown in 
Fig. 1, except for an object characteristic 
discrimination unit 19, the same reference numerals in 
Fig. 19 denote the same parts as those in Fig. 1, and a 
detailed description thereof will be omitted. 

The object characteristic discrimination unit 19 
discriminates the state of an object on the basis of an 
image stored in the work memory 13, and determines the 
pixel shift photographing method in correspondence with 
the discrimination result. 
[Photographing Operation] 

The photographing operation of the digital camera 
according to the fourth embodiment will be described 
below with reference to the flow chart in Fig. 20. In 
this flow chart, since the processing in steps SlOO to 
S106 and step S112 is the same as that in Fig. 3, a 
detailed description thereof will be omitted. 

Normal photographing is done in step S1107 to 
discriminate the state of the object to be photographed, 
and the object characteristic discrimination unit 19 
discriminates the state of the object on the basis of 
the photographed image in step SI 108. In step SI 109, 
pixel shift photographing is done on the basis of the 
discriminated state of the object. Note that the 
operation of normal photographing in step S1107 is the 
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same as that in steps S200 to S206 in the flow chart 
shown in Fig. 4, i.e., an image for one frame is 
photographed and is stored in the frame memory. 
[Object Characteristic Discrimination] 

The object characteristic discrimination method 
will be explained below with reference to the flow 
charts in Figs. 21A and 2 IB. 

In step S1250, green data of image data taken by 
normal photographing is transferred to the work memory 
13. In order to detect frequency components contained in 
this green data, the green data for one frame is 
segmented into 8x8 pixel blocks and is converted into 
two-dimensional frequency data by DCT in units of blocks 
in step S1251. Note that the present invention is not 
limited to DCT as long as green data can be converted 
into two-dimensional frequency data. For example, other 
orthogonal transforms may be used or wavelet transform 
may be used. 

The frequency components of the green data are 
calculated in units of blocks in step S1252, and it is 
checked in step S1253 how many components equal to or 
higher than a predetermined frequency are contained. For 
example, the total of the absolute values of DCT 
coefficients with i > 4 and j > 4 in an 8 x 8 pixel 
block shown in Fig. 10 is calculated, and the totals of 
the absolute values of DCT coefficients calculated in 
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units of blocks are summed up, thus easily detecting the 
ratio of high-frequency components contained in the 
photographed image. 

On the other hand, the total of the absolute values 
of DCT coefficients with i > 4 and j = 1, and the total 
of the absolute values of DCT coefficients with i = 1 
and j > 4 are calculated, and the calculated totals of 
all blocks are summed up, thus also easily detecting the 
ratio of high-frequency components contained in the 
photographed image. 

The sum is stored in an internal register SUMG of 
the CPU 1 in step S1254, and frequency analysis of red 
data contained in the photographed image is started in 
step S1255. DCT coefficients are calculated in units of 
blocks in step S1256, and the totals of the absolute 
value of DCT coefficients equal to or higher than the 
predetermined frequency are calculated in units of 
blocks as in the green data in step S1257 . The sum total 
of the absolute values of DCT coefficients of all blocks 
is calculated in step S1258, and is stored in a register 
SUMR in step S1259 . 

In case of red data, as can be seen from the color 
filter array on the imaging element shown in Fig. 2, 
since the number of pixels is smaller than that of green 
data, red data of the respective pixels are calculated 
by interpolation using surrounding pixels, and 
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high-frequency components attenuate accordingly due to 
the low-pass filter effect. Hence, in case of red data, 
the sum total of the absolute values of high-frequency 
components of all blocks calculated by the same method 
as the green data is multiplied by a predetermined 
coefficient (> 1) . and the product is stored in the 
register SUMR, in place of detecting high-frequency 
components contained in the photographed image by the 
same method as green data. 

As another method, high-frequency component 
information obtained from the red data can also be 
corrected by calculating the total of the absolute 
values of DCT coefficients within a frequency-component 
range broader than that of the green data, e.g., within 
the range i > 3 and j > 3 . 

As for blue data, the same processing as that for 
the red data is executed in steps S1260 to S1264, and 
the result is stored in a register SUMB. 
[Determination of Pixel Shift Method] 

In step S1265 and the subsequent steps, the three 
pieces of high-frequency component information of the 
green, red, and blue data stored in the registers SUMG, 
SUMR, and SUMB are compared to detect a color component 
that contains a largest number of high-frequency 
components, and a pixel shift method optimal to that 
color component is determined. In the following 
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description, the high-frequency component information of 
the green, red, and blue data stored in the registers 
SUMG, SUMR, and SUMB may often be called by the same 
name as the register. 

SUMG and SUMR are compared in step S1265, and SUMG 
and SUMB are compared in step S1266. As a result, if 
SUMG is larger than SUMR and SUMB, i.e., SUMG is 
largest, a largest number of high-frequency components 
are contained in the green data of the photographed 
image. In this case, the drive amount of the optical 
unit 4 upon pixel shift photographing is determined by 
the processing in steps S1267 to S1272. 

A drive amount +P for moving the optical unit 4 by 
an amount corresponding to a pixel pitch P of the 
imaging unit 6 is set in AX(1) in step S1267, AY(1) = 0 
in step S1268, AX(2) = -P/2 in step S1269, AY(2) = +P/2 
in step S1270, AX{3) = -p in step S1271, and AY(3) = 0 
in step S1272. Note that the actual drive amount of the 
optical unit 4 and the moving amount of an image on the 
imaging plane have a predetermined relationship, but the 
following explanation will be given assuming that the 
drive amount and moving amount are equal to each other, 
for the sake of simplicity. 

On the other hand, if it is determined in step 
S1265 or S1266 that SUMG is smaller than SUMR or SUMB, 
the red or blue data in the photographed image contains 
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a largest number of high-frequency components. In this 
case, the drive amount of the optical unit 4 upon pixel 
shift photographing is determined in steps S1273 to 
S1278. For example, Ax(l) = +p is set in step S1273, 
AY(1) = 0 in step S1274, Ax(2) = 0 in step S1275, AY(2) 
= +P in step S1276, AX(3) = -P in step S1277, and AY (3) 
= 0 in step S1278. 

In this manner, the pixel shift amounts Ax(l) , 
AY(1). AX(2), AY(2), AX(3), and AY(3) are uniquely set 
on the basis of the comparison result of the 
high-frequency components of the green, red, and blue 
data, thus ending object characteristic discrimination. 
[Pixel Shift Photographing] 

Upon completion of the object characteristic 
discrimination, pixel shift photographing is started in 
step S1109 in Fig. 20. 

In case of the pixel shift method shown in Fig. 7, 
the resolution of green pixels lined up at pitch P in 
both the X- and Y-directions is doubled in both the 
X- and Y-directions, and the resolution of red and blue 
pixels lined up at pitch 2P in both the X- and 
Y-directions can be remarkably irrproved in the 
X-direction but is not so improved in the Y-direction. 
Therefore, the pixel shift method shown in Fig. 7 is 
used when it is discriminated in object characteristic 
discrimination that the number of high-frequency 
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components of the green data is largest. On the other 
hand, if it is determined as a result of object 
characteristic discrimination that the high-frequency 
components of the red or blue data are more than those 
of the green data, pixel shift photographing is done by 
a pixel shift method shown in Fig. 22. 

Referring to Fig. 22, by decentering the optical 
unit 4 by a predetermined amount in the X-direction, the 
first shift image 1702, which has been shifted by one 
pixel pitch P from an initial image 1701 in the 
x-direction, is obtained. Fur therroore , by decentering 
the optical unit 4 in the first shift state by a 
predetermined amount in the Y-direction, the second 
shift image 1703, which has been shifted by p from the 
first shift image 1702 in the Y-direction, is obtained. 
Moreover, by decentering the optical unit 4 in the 
second shift state by a predetermined amount in the 
X-direction, the third shift image 1704, which has been 
shifted by -p from the second shift image 1703 in the 
X-direction, is obtained. 

In this way, by combining a total of four images, 
i.e., the initial image 1701 and three images shifted by 
predetermined pixel pitches from the initial image 1701 
in units of frames, red and blue pixels can be evenly 
distributed in the X- and Y-directions , as shown in 
Fig. 23. 
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The left matrix in Fig. 23 indicates a pixel data 
array obtained after pixel shift photographing. In 
Fig. 23, since green pixel data overlap each other at 
identical positions as compared to the pixel array of 
the imaging element with the Bayer pattern shown in 
Fig. 2, the resolution of the green data is not largely 
improved compared to the pixel shift photographing shown 
in Fig. 9. 

In order to obtain RGB data of the respective 
pixels shown in the right matrix in Fig. 23 from the 
pixel data shown in the left matrix in Fig. 23, a 
central interpolation filter in Fig. 23 is used. 
However, since green pixel data overlap each other at 
identical positions, these data must be averaged in 
advance . 

For example, RGB data corresponding to pixel a 
shown in Fig. 23 are obtained by multiplying data of 
pixel a and its eight surrounding pixels within the 
range bounded by the dotted line by coefficients of the 
interpolation filter. However, since no pixel data are 
present around pixel a, RGB data corresponding to pixel 
data are obtained by multiplying the data of pixel a by 
a coefficient "1" of the interpolation filter. 

On the other hand, RGB data of a pixel 
corresponding to position b are obtained by multiplying 
data of eight pixel data around position b by 
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coefficients of the interpolation filter. Since there is 
no pixel data actually photographed at position b, RGB 
data of the pixel corresponding to position b are 
obtained from the right and left neighboring pixel data 
of position b. Likewise, RGB data of a pixel 
corresponding to position c are obtained by multiplying 
data of eight pixel data around position c by 
coefficients of the interpolation filter. Since there is 
no pixel data actually photographed at position c, RGB 
data of the pixel corresponding to position c are 
obtained from the upper and lower neighboring pixel data 
of position c. 

[Difference in Photographing Characteristics Depending 
on Pixel Shift Methods] 

In this embodiment, the high-frequency component 
information of the image to be photographed, in other 
words, edge information, is analyzed, and pixel shift 
photographing is done by switching the pixel shift 
method in correspondence with the analysis result. The 
difference in image capture characteristics depending on 
the pixel shift methods will be explained below. 

Figs. 24A to 24D are views for briefly explaining 
the pixel shift method shown in Fig. 7. The pixel shift 
method shown in Fig. 7 is effective for the image to be 
photographed that contains many green high-frequency 
components, i.e., edge information. 
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Fig. 24A shows the state before pixel shift, and 
the respective pixels are lined up in a Bayer pattern. 
Gl and G2 indicate green pixels, and R and B 
respectively red and blue pixels. 

Fig. 24B shows the process for shifting pixel Gl by 
one pixel pitch P in the X-direction, by -P/2 in the 
X-direction and P/2 in the Y-direction, and by -P in the 
X-direction. Figs. 24C and 24D respectively show the 
shift processes of pixels R and B by the same pixel 
shift procedure in Fig. 24B. 

As can be seen from these figures, this pixel shift 
method is characterized in that the number of pixels is 
doubled in the X- and Y-directions as compared to that 
shown in Fig. 24A since pixel Gl is placed among pixels 
in the pixel array shown in Fig. 24A. As for pixels R 
and B, the number of pixels is doubled in the 
X-direction, but the pixel distribution becomes laneven 
in the Y-direction. 

Hence, the pixel shift method shown in Figs. 24A to 
24D (or Fig. 7) is effective for an image to be 
photographed, which contains many green edge 
information, and is inappropriate when the image 
contains many red or blue edge information. 

The pixel shift method effective for an image to. be 
photographed which contains many red or blue edge 
information will be ejcplained below with reference to 
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Figs. 25A to 25D, Figs. 25A to 25D are views for briefly 
explaining the pixel shift method shown in Fig. 22. 

Fig. 25A shows the state before pixel shift as in 
Fig. 24A, and the respective pixels are lined up in a 
Bayer pattern. Fig. 24B shows the process for shifting 
pixel Gl by P in the X-direction, by P in the 
Y-direction, and by -P in the X-direction. Figs. 25C and 
25D respectively show the shift processes of pixels R 
and B by the same pixel shift procedure as in Fig. 25B. 

As can be seen from these figures, this pixel shift 
method is characterized in that the resolution is 
doubled without making R and B pixel distributions 
uneven. As for pixel G, the resolution is doubled, but 
since double shift pixel Gl nearly overlaps the position 
of pixel G2, improvement in resolution by pixel shift is 
smaller than pixels R and B. 

Hence, the pixel shift method shown in Figs. 25A to 
25D (or Fig. 22) is not suitable for an image to be 
photographed which contains many green edge information, 
but is effective for an image that contains many red or 
blue edge information. 

To restate, according to this embodiment, an 
imaging apparatus and method which detect the 
characteristics of an object to be photographed, set a 
photographing method corresponding to the detected 
characteristics, and can capture a high-resolution image 
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with low cost, can be provided. 

Note that the setting method of the photographing 
method in correspondence with the object characteristics 
of this embodiment can be applied to 

"photographing- storage mode 2" in the first to third 
embodiments described above. 

As many apparently widely different embodiments of 
the present invention can be made without departing from 
the spirit and scope thereof, it is to be understood 
that the invention is not limited to the specific 
embodiments thereof except as defined in the appended 
claims . 
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